Thinking a bit more about this, do we need a separate sanitizer build? Why not just run all of the CI with a sanitized build?

Thinking a bit more about this, do we need a separate sanitizer build? Why not just run all of the CI with a sanitized build?

It takes forever... we usual test 10k events, I tried with 7.5k for the ecal pn, and was killed at 6 hours. I'm now converging to the point of only testing 1500 events which will take about an hour. (But for some reason I cant keep the ASAN output in the CI)

Thinking a bit more about this, do we need a separate sanitizer build? Why not just run all of the CI with a sanitized build?

It takes forever... we usual test 10k events, I tried with 7.5k for the ecal pn, and was killed at 6 hours. I'm now converging to the point of only testing 1500 events which will take about an hour. (But for some reason I cant keep the ASAN output in the CI)

This sounds really really strange. The sanitizers should have an extremely small impact on performance

Thinking a bit more about this, do we need a separate sanitizer build? Why not just run all of the CI with a sanitized build?

It takes forever... we usual test 10k events, I tried with 7.5k for the ecal pn, and was killed at 6 hours. I'm now converging to the point of only testing 1500 events which will take about an hour. (But for some reason I cant keep the ASAN output in the CI)

This sounds really really strange. The sanitizers should have an extremely small impact on performance

Hmm, I dont know, here is the action that was killed after 6 hours https://github.com/LDMX-Software/ldmx-sw/actions/runs/11007030465 if it helps

Ok the output is being saved now
https://github.com/LDMX-Software/ldmx-sw/actions/runs/11038784203/artifacts/1978699120
but it still doesnt have what I want... I'll play with it more later

(sorry I'm going back and forth between draft and ready -- that's my only way to trigger tests until it's merged [after that we'll be able to do this from the Actions with the dispatch options])

Hmm, I dont understand, in local the ASAN output is in stderr. But in the action I only get

https://github.com/LDMX-Software/ldmx-sw/actions/runs/11044190876

denv fire /home/runner/work/ldmx-sw/ldmx-sw/.github/validation_samples/ecal_pn/config.py 
==5==LeakSanitizer has encountered a fatal error.
==5==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==5==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
error: Recipe `fire` failed on line 95 with exit code 1

even tho LSAN_OPTIONS=verbosity=1:log_threads=1 are enabled... (which btw I dont need when I run locally)

That seems strange... But the simulation finishes and the leak sanitizer issue is only at the end? I'm still a bit skeptical to the value of running leaksanitizer in CI for us so I might just turn it off isntead

But the simulation finishes and the leak sanitizer issue is only at the end?

I'm running ASAN + UBSAN, and they somehow invoke LSAN too. And yes, they only show up in the end, bc of the setting to dont stop on issues (which is a default set by you @EinarElen I believe, but correct me if I'm wrong [given that we fixed all the issues, it's hard to tell]).

I can be convinced either way about the leak part. But the issue about github dealing with stderr from ASAN/UBSAN is not going to be connected to the leak part. Maybe @tomeichlersmith has an idea about the environment settings on the machines used by the actions.

The runners do not swallow anything from stdout or stderr, but they do not have a TTY and so programs that are "smart" and disable certain behaviors in a non-interactive environment (mainly determined by detecting if a TTY is present) will disable those behaviors.

You can mimic running without a TTY to see if that is the issue.

https://superuser.com/a/1430883

LSAN is a part of ASAN so it gets enabled by default. You turn if off with ASAN_OPTIONS=detect_leaks=0

You can mimic running without a TTY to see if that is the issue.

so none of this managed to reproduce what I see in the actions, but I think I found a workaround with specifying the ASAN output, I'll test that now

OK I think I'm ready now, this is the module operandi I propose: we dont save the log for now (it didnt work even with specifying the output for ASAN), we dont run the leak part --> this means currently there is no issues. Then we run this every time a new PR comes in, and if any of those introduce a problem, this test will fail. It will not show what the issue is (given the output issue), but it will show that there is an issue, so we can check out the branch locally and run this and we'll see the problem.

ok so w/o the the leak part, it doesnt actually take extremely long: 1500 events took 30 min, so the 10k would take 3.3 hours. That's still more than the usual 1.5-2h time. I could increase this to 7500. I think the ecal_pn at this point runs all kinda processors so we are safe to just run that with ASAN/UBSAN.

OK, tests are fine, @tomeichlersmith @EinarElen I wont push anymore, please approve if you agree with the changes